CEA’s events team: capacity building and mistakes

Posted 5 November, 2021 (updated 5 November, 2021)
Events enable attendees to make new connections, learn about core concepts, share and discuss new research, and coordinate on projects. This post sums up our work on events over the third quarter of 2021.
(The post was written before EA Global: London 2021, and refers to that event in future tense.)
In summary:
  • We put a lot of effort into building up our team.
  • We ran two well-received medium-sized events.
  • We made some significant mistakes with one of these events, which means that the counterfactual impact might have been negative. We plan to reflect on this after EA Global is complete, and we expect to make significant changes to address this.
  • In the past few weeks (as of 10/26/21), we’ve managed to roughly double the capacity of EA Global: London, in response to an unexpectedly high number of very strong applications. We’re currently focused on this event.
  • Depending on how EA Global: London goes, we expect to increase the total number of new connections (our key metric) relative to previous years. Optimistically, we might double the number of connections relative to 2019 (though the increase will likely be somewhat lower).

Metric: connections

Past surveys (e.g. Open Phil’s survey) suggest that connections between individuals are the key source of impact from our events. So we focus on the number of new connections we make at our events.
Currently, we calculate this as follows: number of attendees x average number of self-reported new connections. We define a new connection as “someone you feel comfortable reaching out to to ask for a favor”, so this is a relatively high bar. We expect that this is an overestimate, since people who engage a lot with the event are more likely to fill in our post-event survey, and also more likely to make lots of connections. However, we think it allows for some comparison between events and years.
We’re currently working on fine-tuning this metric, finding better ways of measuring it, and thinking through other important outcomes to consider for measuring event outcomes.
Broadly, we think that we’re on track to nearly double new connections compared to 2019. This would be a slight increase from 2020. However, we think this methodology somewhat overestimates growth relative to 2019, and underestimates growth relative to 2020:
  • We think this methodology overestimates the number of connections at virtual events relative to in-person events1; all events in 2020 were virtual, and some events in 2021 were virtual, while all 2019 events were in-person.
  • 2020 figures include EAGx, whereas 2019 and 2021 figures don’t.
See footnote #2 for details on how we came up with our "projected" figure.2

Staffing

Overall, we’re currently in a transition period for the team: we’ve made some very strong hires who we are now onboarding, or who are joining later this year. During this onboarding process, we’re still understaffed. But we think the team will be stronger in about 6 months.
Additionally, Amy is transitioning from a role focused on direct work on events to a role more purely focused on management, with support from Max (Executive Director).
  • We hired
  • Barry Grimes is transitioning to the Happier Lives Institute starting in November. We’re grateful for all of his hard work on events over the past 2.5 years.
  • We’re working with Anine Andresen (previously a contractor) to increase her hours, in the hope that she might become a full-time team member.
  • We expect to bring on at least one additional hire in the next 3-6 months. In the meantime, we have hired some additional contractors:
    • Ivan Burduk to support Admissions and Stewardship
    • Ashley Lin to support Event Production, starting in October

Picnics

We ran the EA Picnic in San Francisco (July 11).
  • Outcomes:
    • 191 attendees
    • Attendees who filled out the survey:
      • averaged 4.6 new connections formed through the Picnic
      • rated the Picnic as 8.38/10 on likelihood to recommend (LTR)
  • We considered not running this event due to capacity limitations, but elected to do so anyway as a means of testing a new format. We’re not sure if this was the right call: it meant we were more stretched in the run-up to EA Global and the Meta Coordination Forum.
We supported two similar events on the US East Coast (in NYC on October 2nd, and in Boston on October 9th), since EAGx East Coast was delayed. However, we think the majority of the impact from those events is attributable to their organizers, who don’t work for CEA.

Meta Coordination Forum

In September, we ran the EA Meta Coordination Forum, which had 40 attendees. The event was focused on facilitating ad hoc collaboration between key people working in the EA “meta” space. (This is a successor to past series of Leaders Forum events.)
So far, feedback from attendees has been positive on average:
  • At time of writing, we have 22/40 responses to the survey.
  • Median new connections = 4 (Mean = 5.33).
  • Self-reported comparisons with counterfactual value of time
    • All but 3 people found that the event was a better use of their time than the counterfactual.
    • Most of the respondents (9) told us that the time spent at the forum was 3-10x as valuable as the counterfactual.
    • A rough (and conservative) interpretation of responses gives 10.77x the counterfactual as the average response. This value is similar to that of the last in-person Leaders Forum (2019) and higher than that of the last virtual Leaders Forum (2020).
However, we made some mistakes in the run-up to the event, which meant that some key attendees could not attend (due to US restrictions on who could enter the US). We think that we should have done more contingency planning for the scenario where US restrictions stayed in place longer than we anticipated. We also should have asked attendees for their preferences about timing and location earlier than we did.
We also could have done more to support attendees who were travelling from abroad and dealing with difficult COVID restrictions, and we failed to communicate promptly with some invitees. Less significantly, there were some operational issues onsite: for instance, making snacks easily accessible to attendees and ensuring that food was produced on time.
For this reason, we think that the event was valuable, but that it was probably less valuable (maybe significantly less valuable) than it could have been. We’re still a bit unsure about this, because we think that delaying or moving the event might have caused us to lose different key attendees (and that delays would have delayed some of the learnings/plan changes produced by the event, which might be important).
We think these events are important, and we value attendees’ time very highly. As a result, we plan to reflect carefully on these mistakes, and we are contemplating making some major changes (such as changing who is responsible for the event in the future).

Future events

Most of our focus is on running EA Global: London (in October). We were pleasantly surprised by the number of strong applications. This led us to increase the capacity of the event from around 500 to 1000 in the past few weeks (as of 10/26/21). We also created a concurrent virtual event, which has hundreds of attendees registered. We think that this might roughly double the value of the event, which would be quite exciting. This change does increase the risk of logistical issues, as well as the risk of COVID spread, but we think these risks are outweighed by the additional value, and we’re doing our best to anticipate and mitigate these risks.
The quality of applications also meant that we had to waitlist or reject some very strong applicants, because of limits on how many people we could accommodate even after increasing our capacity. Where possible, we’ve tried to point these applicants to other events or resources, so that they can continue to engage with EA.
We are also supporting EAGx Prague (in December).
The schedule for 2022 is still being developed, but with our increased capacity, we are considering running 3 large EA Global conferences in 2022. Additionally, we will be supporting EAGxBoston, EAGxOxford, EAGxSingapore, EAGxAustralia, and a virtual Student Summit.

Reflections

Overall, the events team is in the middle of a difficult transition, but I expect us to be stronger on the other side.
We’ve had greatly reduced capacity while we focused on hiring and onboarding. We still managed to run multiple events alongside this, but we made some mistakes in those events. We’re hopeful that our investments will pay off, and that we’ll be able to run more and better events in future years.

  1. We expect virtual events to overreport total numbers of connections relative to in-person events. As mentioned in the body, this figure is calculated as the number of attendees x average number of self-reported new connections. And we expect that this is an overestimate, since people who engage a lot with the event are more likely to fill in the survey, and also more likely to make lots of connections. We expect this effect to be bigger for virtual events, because there’s more variation in how much people engage with the event, and because fewer people complete the survey (causing more bias). Therefore, we expect this method to broadly overestimate the number of connections made in 2020 (and to some extent 2021) versus 2019
  2. This assumes 950 people at EAG: London in person and 850 people attending virtually. We also assume that average connection numbers for those attendees are similar to those from past in-person events (8.3) and slightly lower than those at previous virtual events (4, relative to a historic average of 4.4) (respectively). This gives a total of 950 * 8.3 + 850 * 4 = 7885+3400 = 11,285 connections from EAG. We also expect around 200 connections from the Coordination Forum, which adds a total of 11,485 to the 6,144 number above.