A Machine Translation experiment with Slate Desktop

with 2 Comments

Disclaimer: This is a pretty important bit. I was contacted by the CEO of Precision Translation Tools Pte Ltd on 18 May 2017 to participate in their blogging translator review program. I established immediately that I do not endorse or disavow products and that my time to experiment with it was limited, which was accepted.

Ultimately, this resulted in me only using the software for a short fraction of the 45-trial period given. I am also not an MT expert, and just took upon myself to test it where I thought I could leverage the most out of this resource.

Also, for clarification this program offers as an incentive a free license of their Slate Desktop package. Since I was unable to comply with their exact terms of the review program (see final notes), I expect the license will be terminated, and will request for it to be so.

Finally, because the files I work with are confidential in nature, you will not see screenshots of the software results.


What is Slate Desktop

and why does PTT differentiate it from Machine Translation

 

Slate Desktop is a piece of software made by Precision Translation Tools. They differentiate it from machine translation as the general, current concept, but it is essentiallythat: a local, personalized machine translation (MT) engine (or engines, as you can create them to your heart’s content).

I’ve always been generally unimpressed with MT,  but nonetheless found the availability of a solution that ensures confidentiality to be interesting and worthy of exploring.

After having discussed things briefly with PTT, I eventually pulled together my concept of my test project: I’d be throwing it a bundle of documents I translate every six months or so, pertaining to a few clinical trials. Heavily regulated structure and text (more on that later), repetitive texts, for which I usually can leverage my Translation Memories (TM) to make my life easier and the quality of the job higher.

In the end, I processed roughly 15.000 words of text through it, over a period of a week, and this blog post discusses my impressions.

 

Everything has a beginning…

Installation, set-up, and building an engine

 

So, first things first. Installation is a fairly simple affair and you just follow the instructions. There’s a lot of short tutorials available on Slate’s website and the software does deliver on the promise of ease-of-use.

From then on, you’re pretty much set to build your own MT engines. And herein lies the attraction of the software: your own, local MT engines, with confidentiality, based on your own resources. You can translate the files directly using a XLIFF format, or have a plugin to get MT results in memoQ. It currently integrates with Trados Studio and OmegaT, but I went with what I had at hand.

Now, the catch is this: the engine is only as good as your resources, and if you deal with “imperfect” source files, your mileage WILL vary. For the purpose of testing the software, I chose a language pair that provided the ideal type of documents for a workflow where MT could take part: documents translatable directly with a CAT tool – or for which I could obtain good quality OCR – on a highly regulated subject, with reasonably strong TM hits. The hypothesis I wanted to test was whether Slate Desktop would build a coherent segment when the text differed from those in the TM, and if there was any other unforeseen bonus to its use.

Whenever you build or use an engine, there’s a window that gives you information on it. The Engine Summary panel data for my test project was as follows:

Evaluation sentence pairs: 2,238
Evaluation BLEU score: 82.48 (< 1.0: 73.473)
Quality Quotient: 44.77% (correct test sentences)
Edit distance per line (non-zero): 19.9 (Levenshtein)
Built from 18 files with 33,045 phrase pairs.
Tokens for language A: 429,163
Tokens for language B: 439,495
Language model built from 43 files, 33.715 phrases and 503,369 tokens.

 

The first issue I found was that out of 23 TMX files I had selected, only 18 were successfully used by the software. This was certainly a problem caused by me. Still, that yielded a reasonable MT engine to experiment, if weakened by a relatively low count in phrase pairs and a small corpus size (despite totaling 159 MB of TMX files).

I then proceeded to connect Slate to memoQ as an MT service – there’s a plug in for that, and you get the Slate MT results on the same pane as your Patch match, TM and TB results. I have also tried translating an XLIFF file in one go – which works fine, but it is best to stick with the CAT plug in for the reasons I’ll detail below.

 

… a middle…

Initial impressions

 

Well, bear with me – I’m not in any of the opposite camps of the MT pseudo-war . I’m neither a hater nor a fan of any technology. I use whatever works for me, and allows me to work better, faster, and to provide a better-quality service to my clients. And the TLDR for Slate is this: yes, I can incorporate it into my workflow, but maybe not for what I was thinking at first.

 

So, allow me to share the positive aspects of my experiment. For me, Slate was:

  • Almost always better than a <80% fuzzy match in shorter sentences (<20 words). MemoQ does give some legendary funky fuzzy matches with shorter segments, and Slate was quite goo with those.
  • Oddly enough, Slate was also better for those 99% matches due to different punctuation. I would have expected memoQ to realise by now that if the only difference between a segment is the punctuation at the end, it might as well go and fix it automatically.
  • It was often better than Patch Matches, where memoQ occasionally includes words from the source text to fill in the gaps. Mind you, so does Slate, but differently.
  • Slate’s engine provided some interesting, unexpectedly helpful translations for which there was no TM at all (or at least above the threshold set for TM hits).

But, as every coin has two sides, for me, Slate was:

  • Not that great with formatting. Importing an XLIFF file basically meant all formatting and tags were gone.
  • Not hat good with capitalization in memoQ. If the source segment starts with a capital, odds are I’ll need one too in the MT result, rather than everything in lowercase.
  • Not that great regarding the syntax reversal between a Germanic and a Latin language (English and Portuguese). It also tended to flip around individual’s last names, which I found amusing (e.g. “John Alexander Doe” would become “John Doe Alexander”). Terms (e.g. disease names) would sometimes be reversed in order too.

 

My real-world application

 

OK, first of all, it’s nowhere near as bad as it may seem from the “What’s Hot/What’s Not” above. I did use this for actual work, it did speed up my work considerably, and it did surprise me more than once by allowing me to press two keys rather than type or dictate.

The 23 TMX files I employed were specific to a certain process, each of them for a separate clinical trial. Of the 15.000 words that I processed with Slate embedded in the workflow, roughly 8.000 came from PDF files with extractable content, and the 7.000-ish remaining came from scanned documents, which were then subject to OCR and imported into memoQ.

Now, I mentioned two things beforehand: a) the quality of the source files and b) how Slate surprised me with some results.

  • In the set of documents I used for this test, the grammar was in some instances absolutely horrendous – I couldn’t expect any MT engine to make sense of that, as quite often it requires the translator to know what the author meant but didn’t write to convey the actual message. Spelling mistakes also threw Slate off the road, but again, it was a problem with the quality of the source, and the old “garbage in, garbage out” maxim applies. However, had I used these documents to build the TMX files and then to build the MT engine, I expect that could have been even worse.
  • Where it did surprise me was when handling safety reports, namely long, tabulated lists of Severe Adverse Events, classifications, and so on. Where one would expect to leverage from translation memory, the imperfect source kept reversing the order of the SAEs, adding meaningless expressions and a bunch of other gibberish that, at the end of the day, simply didn’t yield a TM hit. Slate ate these for lunch. Other than the capitalization issues (easily fixable via quick keystrokes, but annoying) and the occasional reversal in the order of the technical terms (a bit more annoying), little to no intervention was needed. For anyone who can feel empathy with me regarding these documents, this was a very nice and welcome bonus.

 

Slate also gave me a slightly less pleasant surprise, when perfectly structured sentences were yielded by it, only to be partially filled with words in the source language. These are called “Out Of Vocabulary” words, and Slate’s support site suggests these as good candidates for enforced terminology. As I had it, roughly 15% of my engine had a BLEU score of 0, which in turn caused these issues. I did not have the time to fine tune the engine, but if enforced terminology is used as anchors, maybe it solves the problem.

 

… and an end.

Final opinion:

 

MT is here to stay, and anyone who doubts that is fooling themselves. However, MT is not meant or feasible for every single application, and by ignoring this, most of the time the results we see are comical, if not tragic. The good news is you don’t have to use it. But if you can, building your own MT engine out of your own work may well provide another tool to improve your quality of life as a translator.

Note that I’m not discussing quality, profitability, margins or anything like that. That is not my point, and people can go and have a go at each other’s throats as they always tend to do when this subject is brought out, anyway. Because this engine is “living” on your computer, and is based on your work, don’t expect it to steal your job or outperform you.

Nonetheless, a very important metric for me is my quality of life: anything that can make my professional life easier without compromising any quality factor (and, if at all possible, enhancing them) is worth considering. As you may have read, I have a special interest in speech recognition and the benefits to ergonomics and productivity it brings. In my short and limited experience, Slate does not offer the same level of benefits on its own, but it does provide a nice complement to speech recognition. Inserting perfect MT hits is still faster than dictating. And, borrowing from discussions with David Hardisty and Kevin Lossner, MT can provide a nice framework for you to dictate differently – just look at the MT engine result and dictate, reading what you want to take from it, and overriding what is unusable.

This, for me, would be the main reason to acquire this software. I can imagine it could do better with a vaster, more organized corpus, but I’ll have to defer from affirming that for now. What I can say is that if I had paid for the test license, it would have paid itself this week. And that’s not a bad metric in my book.

 

Slate Desktop costs USD 549, and PTT offers a money-back guarantee and a 30-day trial period, so they’re putting their money where their mouth is.


Final notes:

I was unable to comply with the full terms of the review program for Slate. Therefore, they may avow or disavow this review. As one of the files I would be required to send under their instructions contains content from the documents used to build the engine, under no circumstances will these files ever leave my machine in their present, unredacted form, due to confidentiality issues. In the spirit of full disclosure, I also exceeded the 45-day evaluation period, despite having warned of my limited availability beforehand. An engine review process would have taken place before writing the review, but didn’t. At the time of posting, PTT has extended use of the trial license.

Still, and as per the remaining subscribed terms, do go take a peek at their page here.

2 Responses

  1. […] giving up work isn’t an option How to price editing and proofreading work. Risk versus time A Machine Translation experiment with Slate Desktop Takeaways from Localization World 34 in Barcelona To certify or not: Staying competitive as a small […]

  2. […] Neto posted this article July 7, 2017 on his blog, https://tiagoneto.com/slate-desktop as part of our Blogging Translator Review Program. During our agreed review period, our […]

Leave a Reply