Noms

Noms by Pair

No More Grunt (Knowledge) Work

Drafting of meeting minutes, or Notes of Meetings (NOMs), within the government is a highly nuanced but tedious task. It can take anywhere from a few hours to a full civil servant’s work day. As part of the Pair suite, Noms draws from your meeting agendas, materials and recordings to create Government-style minutes, in days/hours minutes.

Why Meeting Minutes?

With the launching of products such as Pair Chat and Pair Intern, we see a big uptake in LLM-centric products amongst civil servants.

While these products are good for creative and rudimentary tasks such as drafting an initial policy paper or summarising information, there are limitations with addressing tasks with a defined expected output. Part of the problem is that we could not fine-tune the model according to a given task, as there is a lack of input-labeled output pairs or a concrete set of metrics for a given specific domain. That being said, there is one specific use-case we found in the government that fulfils this use case - Government Meeting Minutes.

But really, why Meeting Minutes?

Meetings are an interesting paradox - on one hand they are crucial to an organization. On the other hand, they are painful, tedious, and frustratingly long.

One major pain that comes out of meetings is the writing of meeting minutes, or NOMs. This is work that is synonymous with government officers. It takes up a lot of officers’ valuable time both in and out of work to meet the deadlines.

Thus far, current solutions have focused more on improving the transcription process - a supplementary aid rather than the work at hand. We are not interested in simply converting speech to text. Instead, our focus is the actual work secretariats have to perform. We aim to write public-style meeting minutes as well as the best human officers by:

• Writing good, factual, stylistically consistent minutes

• Processing information captured during the meeting

• Checking it against supporting documents or other forms of “institutional knowledge”

• Coming up with the minutes in the appropriate format and lingo (e.g. Director A states … Deputy Director B to follow up with action …)

When done right, this will not only save the government hundreds of man-hours every week, but allow meetings to be more focused and organized. 

Opportunity and Impact

Noms was prototyped during OGP’s Hack for Public Good 2024 Hackathon. The team has piloted it in meetings within government agencies, and has drastically reduced the time taken to generate minutes by up to 80%. The team has since been asked to provide further use of Noms to support upcoming meetings.

"We would really like to continue using it for our subsequent meetings!" - Public Officer from Ministry

"The generated draft is good and managed to capture specific terminologies used." - Officer from Government Statutory Board

"My sense is that the draft is almost there and that the minutes generated has a good accuracy of what was captured during the meeting!" - Officer from Government Statutory Board

With a high demand from multiple agencies across the government, Noms is expanding its pilot to various other ministries and statutory agencies (e.g. MCI, TTSH, PSD).

Building the Noms

Noms generates meeting minutes through multiple steps: 

• Transcription

• Segmentation

• Contextualisation

• Summarization and styling

The transcription step first makes use of a high-performing, self-hosted variant of OpenAI’s Whisper model to generate a transcript with proper speaker diarization. 

The transcription then goes through a segmentation layer, which allocates different parts of the transcript to the corresponding agenda item. The segmentation step currently assumes that transcript chunks for each agenda item are continuous and independent of one another (ie; agenda items are done sequentially and not mixed around). These assumptions were made partly from the initial user testing sessions we had with some of our agency partners. 

In order to further improve accuracy and to provide additional context to a meeting, the contextualisation layer is added, where specific supporting document pages are correlated to the respective transcription chunk. Contextualisation is done by first indexing the agenda item slides, followed by a probabilistic semantic scoring mechanism that ensures each transcript segment is rightfully pegged to the correct page range of a supporting document.

At this stage, each agenda item is pegged to a transcript, of which transcript chunks are contextualized with the relevant page range in the agenda item’s supporting document. Summarisation and styling is then performed on all this information through prompt engineering to generate the draft meeting minutes.

The prototype was built based on recent strategies in LLM and prompting optimisation. These strategies would allow the entire pipeline to be easily trainable to specific contexts and use cases. In the next few weeks, the team will be looking into better improving the personalisation for Noms, to get your meeting minutes in your style.

Meet the Nomies

The prototype was developed by the following team.

What's Next?

Noms is constantly being enhanced. Some features would include automating factuality checks for the generated meeting minutes, improving referencing to all meeting material uploaded, as well as integrating with the current modes of holding meetings. 

Given the nuanced nature of different agencies, one possible route is developing bespoke minutes generators which differ based on the style and preferences of your organization. The team is also looking into making Noms accessible to higher data classification and sensitivity level meetings. 

Bigger Picture

The bigger picture is that we believe this generation of AI (e.g. LLMs) can actually accomplish knowledge work end to end, which was not possible with previous tech. Work could get simpler, and higher quality. We are doing Pair Noms to build capability to do that – taking in finished work product and input materials, and learning how to tune underlying models to produce the high-quality work we expect to depend on.

If you're currently a secretariat, or have a meeting in mind to try out Noms, sign up for our pilot here!