Counting 3GPP contributions — even the “approved” kind — does not measure 5G leadership and value

It has been a few years since I last discussed how 3GPP contribution counting is a flawed practice and why it should not be used to evaluate a company’s leadership in technology innovations.

The key point is that 3GPP specifications are not based on the direct acceptance or rejection of individual member contributions, but rather the progressive and collaborative refinement of gradually accepted core concepts.

Technical Documents (TDocs) — written contributions to 3GPP meetings — often are simply “noted” even if the concepts they propose are accepted by the working group. This is a very fundamental fact that some in the industry have apparently missed. Other types of technical documents capture discussion outcomes involving multiple companies. At a later stage, typically after a 3GPP Release is “frozen,” Change Requests (CRs) improve the quality of specifications, with only a small subset of them introducing innovations. Simply counting contributions also fails to give credit to those that are building consensus and driving those core concepts toward the final specifications.

Throughout this blog post, we will refer to our previous blog post on contribution counting, what changed in 3GPP since then, how 3GPP operates, and how the more recent trend in “approved contribution counting” is also unreliable.

Taking action to combat contribution counting

In recent years, it has been encouraging to see 3GPP’s acknowledgement of the issues stemming from contribution counting, which we referenced in earlier blog posts. In particular, it ended up being widely noticed that some member companies were artificially inflating the number of their contributions, for example, by splitting one technical contribution into multiple contributions. To mitigate such behavior, 3GPP leadership instituted countermeasures across different working groups. Since 2017, for instance, in 3GPP RAN1, each company is limited to making only one contribution per meeting agenda item. In other words, a de-facto “contribution cap” exists, equal to the number of distinct agenda items. Other working groups later introduced similar rules. These new rules of engagement were effective and virtually put an end to the phenomenon of inflated contribution numbers.

However, outside 3GPP, the practice of counting contributions unfortunately kept evolving. For the rest of this blog post, we will address the new variants that have come to prominence in certain circles.

Contributions do not tell the whole story

Before we further analyze how contribution counting has changed, it is crucial to understand how technical specifications are created inside 3GPP. I have previously talked about this in extensive detail in my blog series, but let me recap below:

• Typically, a group of companies starts a new 3GPP project for a specific technology area (e.g., 5G NR in unlicensed spectrum, 5G for drones, 5G for XR, etc.) by proposing a new Study Item or a new Work Item in the appropriate group. This is usually driven by either a recognized industry need (e.g., improve coverage in certain situations) or when a company, or group of companies, shares the outcome of their early research and leads the entire ecosystem down a new path.
• Once a project is approved, either as a Study Item or as a Work Item, any 3GPP member company can contribute ideas toward it.
• The 3GPP process is such that every individual component of the project is debated, discussed, and decided individually, based on technical merit. This occurs through face-to-face meetings, online meetings, offline discussions, and written contributions. For each project, there are hundreds, if not thousands, of such micro-decisions. Importantly, these micro-decisions are not captured as “approved” documents, as further explained below.
• A specification editor, who can sometimes be the “rapporteur” of the project, or a so-called “feature lead,” captures the collection of agreements from this large number of discussions and incorporates them first into a temporary document. At some point in time, in certain working groups, agreements were captured in documents called “way forward.” This is when some TDocs are actually “approved,”
• Later, those numerous agreements, captured in different ways, are introduced into the technical specification by the editor/rapporteur/feature lead. This is another point in time where TDocs are approved.

What is important to note is that the editor (or equivalent) is often not from the company that provided the most ideas into the project, nor from the company that proposed the project in the first place. Instead, he or she is generally an engineer appointed by the working group chairman based on his or her personal competence in that given domain and his or her ability to moderate complex topics.

Because of this, in 3GPP you might have an “approved” document from a certain source company containing a lot of ideas that have to be introduced in the specification. However, this does not mean that those ideas were from that source company. It may very well be that an engineer from that source company was the editor that collected agreements coming from a plethora of discussions.

The rise of “approved” contribution counting

In my previous series, I discussed how contributions are inherently not created equal, and counting them as the main metric for measuring innovation leadership has some serious drawbacks. For instance, some contributions contain key system design elements, while others are minor corrections to spelling or formatting errors.

More recently, certain proponents have been putting forward a new concept: “approved” contribution counting. According to “approved” contribution counting, one should only count contributions marked as “approved” in official 3GPP meeting minutes, and this would provide a measure of technology innovation or contribution from a given company.

This so-called “variant”, however, is still very flawed. As mentioned already, 3GPP standards are absolutely not built through the accumulation of approved contributions from the companies who bring the key ideas to 3GPP.

As I outlined earlier, it is common practice that all technical contributions are officially “noted” or “not treated” in the meeting minutes. This is because the technical contributions are a written support to discussions and decisions that often take place outside of the written process and are mediated by moderators or feature leads.

Obviously, some concepts from some technical contributions are accepted by the community and advance in the process of specification crafting. However, crucially, even those contributions containing concepts that are accepted by the working group are not formally “approved.”

Instead, it is the specification editor, or feature lead, that collates inputs from various sources to compile a list of agreements. And this list of agreements will later make its way into a final specification. The fact that Company X is the author of those “summaries of agreements” obviously does not mean that Company X is the source of all those ideas: it merely means that an engineer from Company X had the responsibility to keep track of them. This means that the notion of “approved” contributions simply does not track at all which concepts are successful and which are not. To further elaborate on this point, let me give a couple of examples from recent 3GPP meetings.

RAN1#109-e: May 2022

In this recent electronic meeting, there was a total of 2,586 TDocs contributed by participating 3GPP companies. This meeting happened at a time when 5G Advanced Release 18 discussions were just starting, and most contributions contained proposals for a wide range of technical topics. We took a closer examination of the contribution type and status in this meeting, and they are summarized below:

In the first table, it is evident that most contributions are marked “not treated” or “available.” Even though concepts from these documents were agreed by the community, they were not approved, because of the working methodology described earlier. Those “agreed concepts” were captured by the editor (or equivalent), but this is impossible to track from a count of approved contributions.

It is also interesting to inspect which documents were approved. Overall, less than 1 percent of all contributions were approved. Out of the 21 approved contributions, there was one meeting agenda, 19 Liaison Statements (LS) out (i.e., communication to other groups), and one meeting report. In other words, in this meeting, the totality of the approved contributions was of somewhat administrative content and none of the approved contributions was a company submission of a technical idea.

RAN1#96: March 2019

Our second example is from a few years back. This specific meeting took place during the second installment of Release 15. Drop 2 had just been completed in late 2018, and 3GPP was working on completing additional features, which were targeted for the middle of 2019.

In this meeting, there was a total of 2,348 TDocs with 103 marked “approved.” Out of those, 96 were CRs, which represented 93% of all approved contributions. This was to be expected: at that point in time, most of the innovative work had been completed and there was a focus on corrections and bug fixes to stabilize the 5G specifications. Those CRs were thus corrections to existing specifications.

Most of the CRs addressed a technical issue, added a clarification to the text, or fixed a spelling/formatting error. In some cases, an individual CR captured many such corrections in a single document, as compiled by a “CR editor” who, again, was someone appointed by the chairman to consolidate inputs for a specific area that required those corrections. Also in this case, counting approved contributions merely would count a set of corrections to a completed specification, and not how that completed specification came to be in the first place. Moreover, those approved corrections themselves were sometimes an aggregate of other small corrections compiled by an editor.

(In passing, it is also interesting to note that if approved contribution counting were indeed an acceptable measurement of innovation leadership, one could “prevail” simply by submitting the most spelling/grammar corrections to the specifications. But that would obviously be a paradox!)

These are just two recent examples from different phases of the 5G evolution, and there are many others that showcase similar trends.

I do hope that these examples, and the preceding explanation of 3GPP’s working practice, illustrate that approved contribution counting simply does not work as intended by its proponents. The 3GPP standard is not built through accumulation of approved contributions and therefore any count of approved contributions is inherently misguided and will paint a misleading picture.

How do we gauge innovation leadership?

At Qualcomm, we focus on the quality rather than the quantity of 3GPP contributions.

Assessing leadership requires in-depth research on a feature-by-feature basis and a historical understanding of what eventually makes it into the final standards. One should know what was contributed, and when, and compare that with the final outcome in the technical specifications.

We believe this would be the way to proceed in order to assess 3GPP technology leadership.

What’s next?

The work on 5G Advanced has just begun in 3GPP, starting in Release 18 and it is expected to continue in Release 19 and beyond. There is a rich roadmap of technologies that will bring enhanced end-to-end 5G system capabilities to enable new levels of performance and efficiency.

I am personally very excited about the 5G Advanced evolution that will unlock new possibilities and deliver on the full 5G promise. Stay tuned to hear more from us.

Footnote: This understanding has also been corroborated in the literature a few years ago, please see this paper (even though some of the analysis is partially outdated nowadays).