IETF 123 post-meeting survey

The results of the post-meeting survey are now available on a public report. This is a poor quality replacement for the previous interactive dashboards as the survey platform no longer supports those. We hope to switch survey platforms at some point to address this. Thank you to all of you who responded to this survey as we use your views to continually adjust the meeting experience.

Analysis

We received 354 responses, 308 onsite, and 46 remote. With 1738 registered participants, this gives the survey a maximum margin of error of +/- 4.65%.

The results for satisfaction questions include a mean and standard deviation using a five point scale scoring system of Very satisfied = 5, Satisfied = 4, Neither satisfied nor dissatisfied = 3, Dissatisfied = 2, Very dissatisfied = 1. While there’s no hard and fast rule, a mean of above 4.50 is sometimes considered excellent, 4.00 to 4.49 is good, 3.50 to 3.99 is acceptable and below 3.50 is either poor or very poor if below 3.00. The satisfaction score tables also include a top box, the total of satisfied and very satisfied, and a bottom box, the total of dissatisfied and very dissatisfied, both in percentages. Please note that a small number of questions are on a four point scale.

Question changes since the last survey

A new question was added about visas.

Actions taken following the last survey

For this meeting, we made the following changes, prompted by survey feedback:

• Directional signage was again increased to guide participants through the distributed space and new large signs were added showing the entire programme and floor plan.

Satisfaction

Overall satisfaction is 4.52, an excellent result. With some key exceptions, the satisfaction scores remain high.

The table below shows the satisfaction scores for the last six meetings, along with colour coded indicators for the five point scale above: excellent (🔵), good (🟢), acceptable (🟡), poor (🔴), very poor (⚫️)

Success stories

Overall

The overall satisfaction score for this meeting was excellent, the first time we have reached that level. From the data, much of this is down to the choice of venue, hard work of the team, and the many changes we have made in response to feedback.

Venue

There are quite a few positive comments about the venue and multiple requests that we return, reflecting a good satisfaction score. However this is almost identical to the score for the venue at IETF 122 Bangkok so it remains to be seen from the data if this is quite the exceptional venue as it appears from the comments.

Timing of lunch

Shifting the timing of lunch to 1 pm was a big success as shown in both the scores and the comments, and we will consider making that a permanent change.

Farewell reception and Friday afternoon finish

There are still complaints about the Friday finish in the comments but the score for a 4:30 pm finish remains in the general range of most of the satisfaction scores, whereas a 5:00 pm finish was noticeably lower. This indicates that Friday afternoon sessions are acceptable to the majority provided they do not go on too late.

The new farewell reception that follows the sessions on Friday gets a consistently higher score than the welcome reception indicating that this is a successful addition to the program.

Issues that need addressing

Cost of registration

Multiple participants have stated in the comments that the cost of registration is too high and difficult for them to afford and they would like to see it drop significantly. While the data so far is only anecdotal, these concerns do appear to be increasing as global economic conditions deteriorate. We partially test this with Q5 where remote participants are asked why they did not participate onsite and travel/time/accommodation come out the largest barriers.

The IETF is a not-for-profit that relies solely on donations, sponsorship, and meeting registrations fees to operate. It does not have any form of membership fees that can subsidise meetings, but then equally that means that the only time people pay is for meeting registration.

IETF Meetings regularly run a deficit, which means that we can only reduce the registration fees by reducing costs, potentially significantly. However, that can lead to a lower quality service, which in turn would affect the productivity of the meetings and so needs to be avoided. Rather than go down this route, our effort is going into finding donors who would be willing to sponsor onsite fee waivers (or reduced fee options). If you know of any company or foundation that could support this, then please get in contact.

Temperature and ventilation in meeting rooms

We regularly receive complaints about the heating in rooms, generally that they are too cold but occasionally that they are too warm. We ask all our venues to set the temperature in the meeting rooms to 22℃ but for large rooms this inevitably leads to cold and warm spots. We will now experiment with a temperature of 23℃ to see if that improves comfort.

Additionally, we have more people using CO2 monitors leading to requests for greater ventilation, which is something that venues have not adapted to. We continue to work to understand how venues adjust ventilation and educate them about CO2 levels, but it may be some time before improvements are noticeable.

Vegetarian options and food generally

The comments on vegetarian options have been noted and we will shift the balance of vegetarian vs non-vegetarian options. We will also include pre-inspection of food labelling as part of our initial site visits and add a requirement for large, clear labelling into our contracts. We are however constrained by local laws as to what we can ask for on labels and so may not be able to achieve the full clarity that people expect.

Food generally is an issue that always produces strong comments, and in this case the satisfaction score shows a big drop from IETF 122 Bangkok. Much of this comes down to the different approaches that venues take, hotels in particular, to what food they provide and how they provide it.

Side meetings

The feedback around side meetings is consistent, that they need better management and need to be considered at least semi-official. Prior to this meeting the Internet Engineering Steering Group (IESG) and meetings team reviewed side meetings going back the last few meetings to try and categorise them and will repeat that process after the IETF 123 meeting, with a view to developing a more nuanced approach for the different categories of side meetings.

Wi-Fi

The satisfaction score for the onsite network dropped and this was backed up by comments from participants, but we again had the same problem of very few support calls coming through that would have enabled any onsite diagnostic to take place. It is standard IETF NOC practice to scrub all detailed data at the end of the meeting to preserve participant privacy making after-the-fact diagnosis impossible. These issues may have been related to the switch to an IPv6-Mostly network at this meeting, but there is no data to support that one way or another. The IETF NOC will discuss this and report back.

T-Shirts

Unfortunately we had a problem with t-shirt supply for which we apologise that meant that 10 people who were guaranteed t-shirts did not receive them on the day and we are dispatching t-shirts by post to those people.

In order to meet print deadlines, we only guarantee t-shirts for those that register before a published cut-off date and then estimate numbers after that, with a bias towards reducing waste by not over-ordering. This was the first time that our estimation was too low.

And finally

Thank you everyone who responded to this survey, your feedback is much appreciated.