Sentence segmentation is a fundamental process in natural language processing. It involves breaking down a given text into individual sentences, a task that finds applications in various contexts. Whether you need to split a paragraph into sentences for further analysis or present sentence boundaries in a user-friendly frontend application, sentence segmentation is crucial.

At first glance, identifying sentence boundaries might seem straightforward – just look for a period or full stop. However, it quickly becomes complex when you consider cases where a period is used for abbreviations such as "Dr." or in numerical values like "3.14." This simple punctuation mark doesn't always signal the end of a sentence.

In many languages, the period isn't the standard sentence delimiter. For instance, Hindi uses a unique character called the 'danda' sign (।) to indicate the end of a sentence. Additionally, sentence segmentation must account for periods inside quotations, which don't denote sentence boundaries, or periods within email addresses, which certainly aren't sentence endings.

The challenge lies in finding sentence segmentation libraries that can handle these complexities across a wide range of languages while addressing language-specific intricacies. Most existing libraries fall short in this regard. They tend to focus on English or support only a limited number of languages. Furthermore, they may not be adequately performant or actively maintained.

Here at the language team, we needed a robust sentence segmentation library for our machine translation system and our section translation project. The former is a Python project, while the latter is a Node.js project. After a thorough examination of existing libraries, we decided to create our own – not just one, but two libraries: one in Python and another in JavaScript, both serving the same purpose.

sentencex

We are introducing 'sentencex' library, available in both Python and JavaScript. This remarkable sentence segmentation library boasts extensive language support while emphasizing speed and practicality. When it comes to the balance between linguistic precision and versatility for various applications, sentencex prioritizes usability.

In situations where ambiguity arises and linguistic insight is needed, our library errs on the side of caution, avoiding unnecessary splits rather than risking incorrect segmentations. Our performance benchmarks reveal that this library not only delivers impressive speed but also excels in evaluation datasets.

The name sentencex stands for sentence extraction or sentence✂️

Our overarching goal is to provide support for all languages that have a presence on Wikipedia. Instead of defaulting to English for languages not explicitly defined in the library, we've implemented a fallback chain mechanism. This means that the closest language included in the library will be utilized. We've defined fallbacks for approximately 244 languages, and we also have abbreviation data available for around 30 languages.

sentencex python library

Souce code: github repository. Please refer the documentation for usage examples
Python package: sentencex
Demo: https://wikimedia.github.io/sentencex/

sentencex js library

Souce code: github repository. Please refer the documentation for usage examples
NPM package: sentencex
Demo: https://wikimedia.github.io/sentencex-js

This library is already in use in MinT project. It is also replacing the minimal sentence segmentation libray we had in CX-cxserver project. As we start using it in more project, we hope to support more languages and existing languages better.

Section translation is a mobile first adaptation of ContentTranslation tool. It helps editors to translate sections from a source article to its corresponding article in another language using easy to use UI in a mobile interface. Translating content involves many steps such as choosing the right article, languages, sections to translate, cross checking with the existing article, selecting sentence, translating with the help of Machine translation, editing it and finally publishing. Designing and building such a complex workflow in the small mobile screen is a very challenging project.

This project started about the same time Wikimedia Foundation chose Vue as its frontend library. So the language team decided to build this application as a Vue SPA delivered through a MediaWiki SpecialPage. This was also one of the early Vue applications that used a build step rather than delivering Vue compiler to the browser. Vue 2 was the major Vue version at that time. We used Vue-CLI for building, Vuex for state management, Vue-Router for routing, Vue-banana-i18n for i18n. There was no Vue UI library available at that time, so we build UI components based on our demands and adhering to WMF design guidelines. We had a Hot Module Reloading feature by using webpack HMR too. This overall architecture helped us to build the application relatively faster and with a better developer experience.

Section translation was migrated to Vue 3 along with other library upgrades recently and is now in production. In this blog post we are documenting the process we followed.

We had these objectives:

• Upgrade Vue to 3.x
• Upgrade Vuex to 4.x
• Upgrade Vue router to 4.x
• Upgrade Vue banana i18n to Vue 3 version
• Explore replacement of Vue-cli/webpack with Vite
• Integrate HMR with the new tooling

Vite replaces Vue-CLI

We started with replacing Vue-CLI with Vite without upgrading Vue. This was easy, but our jest based unit tests did not work well with Vite. So it was disabled temporary till we replace Vue 2 in Vue 3 in later step. Vite requires an explicit .vue file extension for Vue files, but for now, instead of fixing it, we used resolve.extension configuration in vite.config.js. We were also using @ alias for top level src directory for avoiding relative file imports. Both of these were supported in Vite with the following configuration

resolve: {
  alias: [
    {
      find: "@",
      replacement: path.resolve(__dirname, "src")
    },
    {
      find: /~(.+)/,
      replacement: path.join(process.cwd(), "node_modules/$1")
    }
  ],
  // FIXME: Avoid this configuration and change files to use .vue extension.
  extensions: [".mjs", ".js", ".ts", ".jsx", ".tsx", ".json", ".vue"]
},

It is interesting to notice, that this migration to Vite, decreased the build time by a factor of 10.

Getting Hot Module Reloading working

Since we are going to change lot of source code in the migration it was important for us to retain the Hot Module Reloading very early to save developer time. Vite Dev server runs at http://localhost:3000 and uses Websockets for HMR. Wiring this HMR through ResourceLoader to get it working in a MediaWiki Special page required some exploration, but we got it working. We introduced a configuration to indicate whether in development mode or production mode. In development mode, the main Resource loader module need to be swapped with http://localhost:3000/src/main.js. This is done by a ResourceLoader packagefile callback function. But since http://localhost:3000/src/main.js is an ES module, it cannot be loaded using Resource loader. To avoid this issue, we create a simple js file that does import call like this:

// Doing imports like this overcomes the limitation of resource loader about ES Module imports
import("http://localhost:3000/src/main.js").catch((err) => {
  console.error(
    "Dev server connection failed. Please check if vite dev server is running."
  );
  console.error(err);
});

This main.dev.js is used instead of http://localhost:3000/src/main.js in packagefile callback:

packageFiles": [
				{
					"name": "dist/cx3.es.js",
					"callback": "ContentTranslation\\Hooks::devModeCallback",
					"callbackParam": [
						"dist/cx3.es.js",
						"src/main.dev.js"
					]
				}
			],

This is the packagefile callback:

public static function devModeCallback( ResourceLoaderContext $context, Config $config, array $paths ) {
		list( $buildPath, $devPath ) = $paths;
		$file = $buildPath;
		if ( $config->get( 'ContentTranslationDevMode' ) ) {
			$file = $devPath;
		}
		return new ResourceLoaderFilePath( $file );
	}

Vue compat build

Even before our plan to migrate to Vue3, we had started to use Vue composition API (it's port to Vue2) to manage the complexity of the application. That helped us in the migration, but still there are so many things to fix to get everything work with Vue 3. So we replaced Vue 2 with Vue 3 compatibility build as a temporary way to get everything working, but allowing us to do iterative migration. Note that we have not merged any code yet, we were doing everything in unmerged commit chains.

Vuex 4

As part of this migration, we also needed to use the Vuex 4 version which is compatible with Vue 3. Luckily, almost all Vuex APIs have remained unchanged from Vuex 3 and the migration process was very smooth. Additionally, Vuex 4 comes with a nice new feature that is very handy for us: the useStore composition function that can be used retrieve the store within the component setup hook.

Vue Router 4

Similarly to Vuex, we upgraded Vue Router to version 4, so that is compatible with Vue 3. Once again, most Vue Router APIs remained unchanged for this upgrade and there were only some trivial modifications needed to complete the upgrade for this package. Finally, Vue Router 4 also provides its own useRouter and useRoute composition functions to access the application router and the current route respectively. These functions are very useful for us, given that we base most of our Vue components on Vue 3 Composition API.

i18n

When it comes to internationalization, Section Translation relies on Vue Banana i18n library. This library was created by @santhosh and is actively maintained by @santhosh himself and the Language team. It is basically a Vue wrapper for Banana i18n library. That means that this library supports Mediawiki Internationalization message system, while at the same time it provides out-of-the-box template directives (v-i18n, v-i18n-html) and a composition function (useI18n) to retrieve the bananaI18n utility object. In this case, we had to upgrade the vue-banana-i18n package from version 1.x (which supports Vue 2) to version 2.x that provides Vue 3 support.

Upgrade went fine but code was broken

Although we upgraded the library, we noticed that the useI18n composition function didn't work as expected and the code was broken in all the places where we retrieved the bananaI18n object, even though we only used it inside composition setup hook. After some research we realized that the reason behind this weird behaviour was that Vue Banana i18n library bundled its own version of Vue in its dist files, and it didn't externalize the Vue dependency. Because of that, the bananaI18n object relied on a different Vue instance leading to this unexpected behaviour. In order to fix this issue, we updated the vite configuration inside the library to avoid bundling Vue as a dependency, and the issue got fixed.

CSS logical properties

The Right to left support in Mediawiki applications is usually done using CSS flipping using CSSJanus. This is happened during the ResourceLoader module request with a given language and direction. Since our development setup does not include ResourceLoader at all we were not using CSSJanus. The section translation application is written using CSS logical properties to support script directionality as our browser support criteria enables it. In production, to bypass CSS flip by ResourceLoader, we use noflip: true option in the module definition in extension.json.

Addressing breaking changes in Vue 3

Plugins

One of most important changes in Vue 3 is the way that application-level functionalities are added to Vue. In Vue 2, application-level functionalities were added by using plugins to register mixins or Vue instance methods. However, Vue 3 brings a new feature that serves as a dependency injection mechanism, called "provide/inject". This feature is also a standard way to provide application-level functionality by making a resource injectable inside Vue plugins.

For this reason, we had to refactor our application plugins, to remove the usage of mixins and use this new "provide/inject" pattern. An important property of the "provide/inject" feature is that "inject" is not available outside the setup hook of the Vue Composition API. For this reason, we had also to make sure that there were no invalid assumptions about the availability of the injected resources inside composables that could be executed outside setup hooks.

v-model usage

Another breaking change we had to deal with was the reworked usage of v-model directive inside templates and the replacement of v-bind.sync. In order to limit the changes for this migration as much as possible, we decided to use the v-model:property syntax both for replacing the occurrences of v-bind.sync directives and old Vue 2 v-model syntax. Restoring the usage of v-model in its simplest form would require renaming the props from "value" to "modelValue" and replacing the @input events with update:modelValue events. Since this is not essential for our application to work properly, we deferred these changes for later, when we would have completed the Vue 3 migration.

Vue.set

A very interesting change in Vue 3 is that Vue.set global function and the "$set" instance method has been removed. This became possible because Vue 3 leverages Javascript Proxies for reactivity (unlike Vue 2 that only used getters/setters). So, in Vue 3 we could (and should) get rid of all Vue.set occurrences and use regular JS assignments instead.

Transitions

In Vue 3 some transition classes has been changed and <transition> elements can only have one single child. Hence, we also had to fix our animation CSS classes and the related issues inside Vue templates.

Template refs, nextTick and slots

Finally, Vue 3 introduces some new syntax for several features. Some of them were affecting Section Translation application, so we had to use the updated syntax for template references, replace the "nextTick" instance method that was used for Vue 2 (using this.$nextTick) with the new nextTick method from the Global Vue API and also replace the deprecated slot="slotName" syntax with the new #slotName syntax. All these changes were easy to find out and fix, given the warnings coming from the Vue 3 compatibility build that we were using during this migration.

Replace Vue Compat Build with Vue 3

Once we fixed all above issues for this migration, we had reached a point where we were actually blocked by the Vue Compatibility build. Although, most issues have indeed been fixed by then, there were some other (v-on.native modifier, $listeners object) that were not supported by the compatibility build, either in the old Vue 2 syntax or the new Vue 3 syntax. This led us to remove the compatibility build and switch to Vue 3 entirely. Given the fact that we didn't care about merging the changes piece-by-piece, it was perfectly ok for us to make this change, even if the application still wasn't working properly on Vue 3. However, the switch to Vue 3 enabled us to resolve the remaining issues, that are listed next.

$listeners

The $listeners object has been removed in Vue 3 and event listeners are now part of the $attrs object. For this reason, we had to remove all the component event propagation with v-on="$listeners" directive and either use the v-bind="$attrs" directive or directly emit the events that should be passed to parent components.

emits option and native click handlers

Vue 3 introduces a new "emits" component-level option, similar to the existing props option. This option is used to define the events that a component can emit to its parent. At the same time, the "v-on.native" modifier has been removed in Vue 3 and Vue will now add all event listeners that are not defined as component-emitted events in the child, as native event listeners to the child's root element. Consequently, we had to explicitly add any custom component-emitted event inside the emits option of each component and remove the .native modifier. This went quite smoothly, since it's a straightforward change.

Unit testing - Jest migration

Vue-CLI had wrapper for unit testing with Jest. With the Vite migration, and Vue 3 upgrade we used the @vue/vue3-jest package. The underlying Vue-testutils package was also upgraded to 2.x versions. The migration is documented well in Vue test utils website. We just followed it and the migration was smooth, but the snapshots generated by this version had some alignment and such minor changes. We had to update the snapshots and make sure no importance changes happening there.

Merging

We had a 29 commits long chain of patches. @santhosh and @ngkountas were reviewing patches each other and improving them as required. Only the last one in the chain passes CI tests. How to merge them? We squashed all of them in to a single large commit. And abandoned all the other commits in gerrit.

Everything went well..wait..

We noticed that certain part of code does not work in production mode while working fine in dev mode. Strangely adding some console statements make it working again. We got puzzled by this behaviour for a while, but looking at the minified code that was supplied by RL gave a clue. RL was minifying the vite generated bundle and was failing with some of the ES features like async. Resource Loader minification yet to support them

We had this in place for Vue2, but somehow missed to retain in the migration process. We quickly fixed it as follows in vite configuration:

esbuild: {
    // Avoid ResourceLoader minification
    banner: "/*@nomin*/",
  },

/*@nomin*/ is a special indicator for Resource Loader. Having that at the top of the file tells it to skip minification.

Left overs

We have been using storybook for the development and demo of UI components. Storybook comes with a long list of dependencies, including react. Migrating that to vite based tooling is currently difficult. The upstream is still working on such tooling integration. For now we just disabled storybook building as we are not modifying the UI components much and probably going to use Codex once it is ready.

We also had an experimental set up for integration test with Cypress and its Vue binding. We disabled that as well since we were not writing any actual integration tests so far.

Testing

The testing was done by language team members in our test instance in wmflabs. Our team did not have a QA person to do detailed testing.

Conclusion

We finished the whole migration in 3 weeks time. We had a couple of minor visual regressions but nothing that impacts the application usage. The developer experience is now much improved, thanks to unbelievably fast HMR using Vite devserver. The build time is also in the range of seconds , thanks to esbuild based building. Removing vue-cli and storybook related dependencies reduced the total space taken by npm modules from 0.5GB to 128MB.

This blog post is authored by @ngkountas and @santhosh