<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi all,<br>
<br>
I am a bit puzzled why
<a class="moz-txt-link-freetext" href="http://www.w3.org/TR/charmod/#sec-stringIndexing">http://www.w3.org/TR/charmod/#sec-stringIndexing</a> is renaming
Unicode Code Points (a clearly defined thing) to Character String.<br>
From my understanding the example in
<a class="moz-txt-link-freetext" href="http://www.w3.org/TR/charmod/#C052">http://www.w3.org/TR/charmod/#C052</a> is not good:<br>
"(Example: the use of UTF-16 in [DOM Level 1])." <br>
<br>
UTF-16 is the encoding of the string and is independent of code
points, units and graphems, i.e. you can encode the same code
point in UTF-8, UTF-16 and UTF-32 which will definitely change the
number of code units and bytes needed. <br>
<br>
While UTF-8 has a variable length of one to four bytes per code
point, UTF-16 and 32 have the advantage of a fixed length. This
means that you can use byte offsets easily to jump to certain
positions in the text. However, this is mostly used internally,
i.e. C/C++ has a dataype widechar using 16 bits as it is easier to
allocate memory for variables. Maybe some DOM parser rely on
UTF-16 internally too, but still count Code Points<br>
<br>
On the (serialized) web, UTF-8 is predominant, which is really not
the question here as the choice between graphems, code points and
units is orthogonal to encoding.<br>
<br>
Regarding annotation, using code points or Character Strings is
definitely the best practice. Any deviation will lead to side
effects such as "ä" having the length 2:<br>
<br>
Using code points:<br>
Java, length():
"ä".length() == 1<br>
PHP,utf8_decode():
strlen(utf8_decode("ä"))===1<br>
Python, len() in combination with decode():
len("ä".decode("UTF-8")) ==1<br>
<br>
Using code units:<br>
Unix wc: echo -n "ä" | wc is 2<br>
PHP: strlen("ä")===2<br>
Python: len("ä")===2<br>
<br>
For the NLP2RDF project we converted these 30 million annotations
to RDF: <a class="moz-txt-link-freetext" href="http://wiki-link.nlp2rdf.org/">http://wiki-link.nlp2rdf.org/</a><br>
It was quite difficult to work with the byte offset given that the
original formats where HTML, txt, PDFs and docx. <br>
<br>
Anyhow, I wouldn't know a single use case for using Code Units for
annotation. I am unsure about Graphems. Personally I think, byte
offset for text is unnecessary, simply because code points are
better, i.e. stable regarding encoding and charset. <br>
<br>
There is a problem with Unicode Normal Form (NF). Generally,
Normal Form C is fine. However, if people wish to annotate
diacritics independently. NFD is needed. <br>
NFC: è <br>
NFD: `e <br>
in NFD you can annotate the code point for the diacritic
separately. However, NFD is not in wide use and the annotation of
diacritics is probably out of scope.<br>
<br>
There is some info in <br>
- the "definition of string" section in the NIF spec:
<a class="moz-txt-link-freetext" href="http://persistence.uni-leipzig.org/nlp2rdf/specification/core.html">http://persistence.uni-leipzig.org/nlp2rdf/specification/core.html</a>
<br>
(yes, we consider moving to a W3C community group for further
improvement) <br>
- Unicode Norm Forms: <a class="moz-txt-link-freetext" href="http://unicode.org/reports/tr15/#Norm_Forms">http://unicode.org/reports/tr15/#Norm_Forms</a><br>
- <a class="moz-txt-link-freetext" href="http://tinyurl.com/sh-thesis">http://tinyurl.com/sh-thesis</a> , page 76 <br>
<br>
On my wishlist, I would hope that the new Annotation standard
would include a normative list (SHOULD not MUST) of string
counting functions for all major programming languages and other
standards like SPARQL to tackle interoperability. When
transfering data, it is important that the other implementation
counts offsets the same way. Listing the functions would help a
lot. <br>
<br>
All the best,<br>
Sebastian<br>
<br>
On 30.04.2015 13:01, Nick Stenning wrote:<br>
</div>
<blockquote
cite="mid:1430391685.1335366.260730957.61BA7DE6@webmail.messagingengine.com"
type="cite">
<pre wrap="">Thanks for this reference, Martin, and thanks for passing this to TAG,
Frederick.
The character model lays out the problems more clearly than I have. It's
clear that recommendation is to use character strings (i.e. codepoint
sequences) unless:
a) there are performance considerations that would predicate the use of
"code unit strings" (I presume interop with existing DOM APIs would also
be a strong motivator)
b) "user interaction is a primary concern" -- in which case grapheme
clusters may be considered
Unfortunately for us, both considerations apply in the annotation use
case.
I'd suggest we schedule a discussion of this issue in an upcoming call.
N
On Thu, Apr 30, 2015, at 02:58, Martin J. Dürst wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hello Frederik,
This is an old, well-known issue. As a starter, please have a look at
what the Character Model has to say about this:
<a class="moz-txt-link-freetext" href="http://www.w3.org/TR/charmod/#sec-stringIndexing">http://www.w3.org/TR/charmod/#sec-stringIndexing</a>
Please feel free to come back again here or contact the I18N WG.
Regards, Martin.
On 2015/04/29 21:45, Frederick Hirsch wrote:
</pre>
<blockquote type="cite">
<pre wrap="">TAG members - has the issue of dealing with symbols vs characters/codepoints come up in TAG discussion?
Any comment/suggestion welcome (I've cross-posted intentionally, please remove recipients if not appropriate.)
Thanks
regards, Frederick
Frederick Hirsch
Co-Chair, W3C Web Annotation WG
<a class="moz-txt-link-abbreviated" href="http://www.fjhirsch.com">www.fjhirsch.com</a> <a class="moz-txt-link-rfc2396E" href="http://www.fjhirsch.com/"><http://www.fjhirsch.com/></a>
@fjhirsch
</pre>
<blockquote type="cite">
<pre wrap="">Begin forwarded message:
From: "Nick Stenning" <a class="moz-txt-link-rfc2396E" href="mailto:nick@whiteink.com"><nick@whiteink.com></a>
Subject: Unicode offset calculations
Date: April 29, 2015 at 4:38:34 AM EDT
To: <a class="moz-txt-link-abbreviated" href="mailto:public-annotation@w3.org">public-annotation@w3.org</a>
Resent-From: <a class="moz-txt-link-abbreviated" href="mailto:public-annotation@w3.org">public-annotation@w3.org</a>
One of the most useful discussions at our working group F2F last week was the result of a question from Takeshi Kanai about how we calculate character offsets such as those used by Text Position Selector <a class="moz-txt-link-rfc2396E" href="http://www.w3.org/TR/annotation-model/#text-position-selector"><http://www.w3.org/TR/annotation-model/#text-position-selector></a> in the draft model. Specifically, if I have a selector such as
{
"@id": "urn:uuid:...",
"@type": "oa:TextPositionSelector",
"start": 478,
"end": 512
}
to what do the numbers 478 and 512 refer? These numbers will likely be interpreted by other components specified by this WG (such as the RangeFinder API), not to mention external systems, and we need to make sure we are consistent in our definitions across these specifications.
I've reviewed what the model spec currently says and I'm not sure it's particularly precise on this point. Even if I'm misreading it and it is clear, I'm not sure it makes a recommendation that is practical. In order to review this, I'm going to first lay out the possible points of ambiguity, and then review what the spec seems to say on these issues.
1. A symbol is not (necessarily) a codepoint
The atom of selection in the browser is the symbol, or grapheme. For example, "ą́" is composed of three codepoints, but is rendered as a single selectable symbol. It can only be unselected or selected: there is no way to only select some of the codepoints that comprise the symbol.
Because user selections start and end at symbols, it would be reasonable for TextPositionSelector offsets to be defined as symbol counts. Unfortunately, most extant DOM APIs don't deal in symbols:
</pre>
<blockquote type="cite">
<pre wrap="">var p = document.createElement('p')
p.innerText = 'ą́'
p.innerText.length
</pre>
</blockquote>
<pre wrap="">3
</pre>
<blockquote type="cite">
<pre wrap="">p.firstChild.splitText(1)
p.firstChild.nodeValue
</pre>
</blockquote>
<pre wrap="">'\u0061'
</pre>
<blockquote type="cite">
<pre wrap="">p.firstChild.nextSibling.nodeValue
</pre>
</blockquote>
<pre wrap="">'\u0328\u0301'
Calculating how a sequence of codepoints maps to rendered symbols is in principle complicated <a class="moz-txt-link-rfc2396E" href="http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries"><http://www.unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries></a> and in practice not completely standardised across rendering engines. It's also (as demonstrated with splitText above) possible for the DOM to end up in a state in which the mapping between textual content and rendered symbols has become decoupled.
2. Combining characters
Some sequences of codepoints can render identically to other sequences of codepoints. For example:
ñ (U+00F1 LATIN SMALL LETTER N WITH TILDE)
renders identically to
ñ (U+006E LATIN SMALL LETTER N + U+0303 COMBINING TILDE)
This is the "combining characters" problem. Some codepoints are used to modify the appearance of preceding codepoints. Selections made on a document containing one of these would behave identically to selections made on a document containing the other, but:
</pre>
<blockquote type="cite">
<pre wrap="">'ñ'.length
</pre>
</blockquote>
<pre wrap="">1
</pre>
<blockquote type="cite">
<pre wrap="">'ñ'.length
</pre>
</blockquote>
<pre wrap="">2
This is not an insoluble problem, as the Unicode specification itself defines a process by which sequences of codepoints can be canonicalised into fully decomposed (aka "NFD") or fully composed (aka "NFC") form. But it's not that simple, because if we specify a canonicalisation requirement for annotation selector offsets, then there may be undesirable performance implications (consider making an annotation at the end of a 100KB web page of unknown canonicalisation status).
3. Astral codepoints and JavaScript
JavaScript's internal encoding of Unicode strings is based on UCS-2 <a class="moz-txt-link-rfc2396E" href="https://en.wikipedia.org/wiki/UTF-16#History"><https://en.wikipedia.org/wiki/UTF-16#History></a>, which means that it represents codepoints from the so-called "astral planes" (i.e. codepoints above 0xFFFF) as two surrogate pairs. This leads to the principal problem that Takeshi identified, which is that different environments will calculate offsets differently. For example, in Python 3:
</pre>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<pre wrap="">len('😀') # U+1F600 GRINNING FACE
</pre>
</blockquote>
</blockquote>
</blockquote>
<pre wrap="">1
Whereas in JavaScript:
</pre>
<blockquote type="cite">
<pre wrap="">'😀'.length
</pre>
</blockquote>
<pre wrap="">2
There are ways of addressing this problem in JavaScript, but to my knowledge none of them are particularly elegant, and none of them will allow us to calculate offsets at the bottom of a long document without scanning the entire preceding text for astral codepoints.
So what does our spec currently say?
The text must be normalized before counting characters. HTML/XML tags should be removed, character entities should be replaced with the character that they encode, unnecessary whitespace should be normalized, and so forth. The normalization routine may be performed automatically by a browser, and other clients should implement the DOM String Comparisons [DOM-Level-3-Core] method.
It's not immediately clear what this means in terms of Unicode normalisation. Following the chain of specifications leads to §1.3.1 of the DOM Level 3 Core specification <a class="moz-txt-link-rfc2396E" href="http://www.w3.org/TR/DOM-Level-3-Core/core.html#DOMString"><http://www.w3.org/TR/DOM-Level-3-Core/core.html#DOMString></a>. The only thing this says about Unicode normalisation is:
The character normalization, i.e. transforming into their fully normalized form as as defined in [XML 1.1], is assumed to happen at serialization time.
This doesn't appear to be relevant, as the meaning of "serialization" in this context appears to refer to the mechanisms described in the DOM Load and Save <a class="moz-txt-link-rfc2396E" href="http://www.w3.org/TR/2004/REC-DOM-Level-3-LS-20040407/"><http://www.w3.org/TR/2004/REC-DOM-Level-3-LS-20040407/></a> spec, and does not refer to the process of parsing an HTML document and presenting its content through the DOM APIs. (I'd be very happy if someone more familiar with the DOM Level 3 Spec could confirm this interpretation.)
For completeness, "fully normalized form" in the XML 1.1 sense <a class="moz-txt-link-rfc2396E" href="http://www.w3.org/TR/2004/REC-xml11-20040204/#dt-fullnorm"><http://www.w3.org/TR/2004/REC-xml11-20040204/#dt-fullnorm></a> would appear to imply full "NFC" normalisation of the document. It is apparent even from the simple examples above that browsers do not apply NFC normalisation to documents they receive from the server.
What should we do about all this?
I've listed above three sources of confusion in talking about offsets. In each case there is tension between what would make the most sense to a user and a pragmatic engineering recommendation that takes into account contingent factors.
Symbols: users can only select symbols in browsers, but as far as I'm aware all current internal DOM APIs ignore this fact. Further, given that determining the symbol sequence for a given codepoint sequence is non-trivial, we probably should not attempt to define offsets in terms of symbols.
Combining characters: user selections make no distinction between combinatoric variants such as "ñ" and "n + ˜", so it would seem logical to define offsets in terms of the "NFC" canonicalised form. In practice, such a recommendation would likely be ignored by implementers (for reasons of complexity or performance impact), and so for the same reasons as in 1) I'd be inclined to suggest we define offsets in terms of the delivered document codepoint sequence rather than any canonical form.
Astral codepoints + surrogate pairs: this is the tricky one. As demonstrated by this page <a class="moz-txt-link-rfc2396E" href="http://bl.ocks.org/nickstenning/bf09f4538878b97ebe6f"><http://bl.ocks.org/nickstenning/bf09f4538878b97ebe6f></a>, this poses serious problems for interoperability, as JavaScript counts a single unicode astral codepoint as having length 2, due to the internal representation of the codepoint as a surrogate pair. As far as I'm concerned we're stuck between a rock and a hard place:
a. calculating offsets in terms of codepoints (i.e. accounting for surrogate pairs in JavaScript) makes interoperability more likely, but could impose a substantial cost on client-side algorithms, both in terms of implementation complexity and performance impact.
b. calculating offsets using native calculations on JavaScript strings is preferable from an implementation complexity standpoint, but as far as I'm aware no other mainstream programming environment has the same idiosyncrasy, thus almost guaranteeing problems of interoperability when offsets are used in both the DOM environment and outside.
In summary, Takeshi raised an important question at the F2F. What do we do about JavaScript's rather unfortunate implementation of unicode strings? I'd be interested to hear from anyone with thoughts on this subject. I imagine there are people in the I18N activity at W3C who would be able to weigh in here too.
-N
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<br>
<br>
<div class="moz-signature">-- <br>
<small>Sebastian Hellmann<br>
AKSW/NLP2RDF research group<br>
Insitute for Applied Informatics (InfAI) and DBpedia Association<br>
Events: <br>
* <b>Feb 9th, 2015</b> <a
href="http://wiki.dbpedia.org/meetings/Dublin2015">3rd DBpedia
Community Meeting in Dublin</a><br>
* <b>May 29th, 2015</b> Submission deadline SEMANTiCS 2015 <br>
* <b>Sept 15th-17th, 2015</b> <a href="http://semantics.cc/">SEMANTiCS
2015 (formerly i-SEMANTICS), Vienna </a><br>
Venha para a Alemanha como PhD: <a
href="http://bis.informatik.uni-leipzig.de/csf">http://bis.informatik.uni-leipzig.de/csf</a><br>
Projects: <a href="http://dbpedia.org">http://dbpedia.org</a>,
<a href="http://nlp2rdf.org">http://nlp2rdf.org</a>, <a
href="http://linguistics.okfn.org">http://linguistics.okfn.org</a>,
<a href="http://www.w3.org/community/ld4lt">https://www.w3.org/community/ld4lt</a><br>
Homepage: <a href="http://aksw.org/SebastianHellmann">http://aksw.org/SebastianHellmann</a><br>
Research Group: <a href="http://aksw.org">http://aksw.org</a><br>
Thesis:<br>
<a href="http://tinyurl.com/sh-thesis-summary">http://tinyurl.com/sh-thesis-summary</a><br>
<a href="http://tinyurl.com/sh-thesis">http://tinyurl.com/sh-thesis</a><br>
</small></div>
</body>
</html>