Apache OpenOffice (AOO) Bugzilla – Issue 88070
[a11y] Notes are not exposed to ATs
Last modified: 2013-08-07 14:44:07 UTC
Steps to reproduce: 1. Open one of the following sample documents: http://bugzilla.gnome.org/attachment.cgi?id=108525&action=view http://bugzilla.gnome.org/attachment.cgi?id=108532&action=view 2. Launch Accerciser and use it to examine both the accessible text for the objects in the document and the accessible hierarchy. Expected results: Each embedded object would be represented both by an embedded object character within the text and by an object in the hierarchy (e.g. an object of ROLE_IMAGE exposed as the child of the paragraph which contains that image). Actual results: Embedded object characters are present, but there are not corresponding accessible objects in the hierarchy. Therefore, ATs know that *something* is embedded but have no way of knowing what that something is and hence cannot provide that information to the user.
As described
The objects in the first document are present in the a11y hierarchy, even though their role is inappropriately mapped to UNKNOWN (which I just fixed in CWS uaa06). However, the notes in the second sample document seem not to get exposed via UNO a11y API at all.
Notes (or better annotations) are not embedded objects. What has changed in OOo 3.0 Beta is that the annotations are made better visible and editable than in OOo 1.x/2.x Currently, we have no a11y API to provide annotations to accessibility tools. Thus, first we have to define how such objects should be provided to an accessibility tool. Due to lack of resources for OOo 3.0 I have to re-target this issue to OOo 3.1
3.2, because it's still unclear how notes should be exposed to AT....
@mt For clarification: Are you waiting on input from us on how they should be exposed, or just need more time to work it out on your end?
both. I sent a note to Pete Brunet to clarify how IA2 would expect it, but it would also be great to get ATK input. I will attach a file describing the situation.
Created attachment 58829 [details] Sample document with annotation, visible in OOo 3, not in OOo 2
Attached is an ODF document with some annotations. OOo 3 can nicely display these annotations on the screen, but it's unclear how to best expose them to AT. Annotations can contain one or more paragraphs, each with its own formatting. Annotations can have different colors for the different authors. If too many annotations are used on 1 page, some annotations are not visible – see 2nd page. Annotations can also be completely hidden – try Menu – View – Notes. In ODF 1.0 and ODF 1.1 an annotation can only be at certain position in the text. In ODF 1.2 an annotation can annotate an arbitrary part of the document's content – see approved ODF 1.2 proposal found at http://lists.oasis-open.org/archives/office/200708/msg00007.html. Questions/Suggestions: - They look like objects. Should they be exposed like shapes in a drawing? When they couldn't have formatted text, I maybe would have suggested to simply expose them as an attribute, but that also wouldn't tell the user that there are objects on the screen showing them... - With ODF 1.2 they can be bound to a range starting in one paragraph, ending in an other - how to expose this relation? - When turned off, there is nothing visible, so I guess nothing has to be exposed. Any thoughts?
In order to get this issue solved in time in OOo 3.2 feedback from the accessibility tools experts respectively from the OOo accessibility API users is needed how notes/annotations are best exposed - see MT's comment from 2008-12-15. I am asking to give your feedback in time, because of the expected time schedule for OOo 3.2. OOo 3.2 release is planned for 2009-09. Thus, I expect the code freeze in 2009-07 and the feature freeze in 2009-05 or 2009-06. I propose to have this issue solved for the feature freeze date. Thus, not too much time to specify how notes should be exposed at the OOo accessibility API, perform the implementation and test it.
Speaking just for myself GNOME deadlines seem to have taken over. Things will calm down in another week or so (I believe) due to code freeze. I'm not sure what will be going on in terms of the OpenSolaris 2009.06 release (Will?). But what is starting to go through my head is this: If we crank out some feedback in order to meet the OOo deadline, rather than really think it through and give it the attention that it deserves, we all may live to regret it. Therefore, I'm wondering if this should be pushed back to 3.3 (or 4.0 if there will not be a 3.3). We should still prioritize giving you the feedback you need -- make it a high priority for as soon as things calm down this spring. Thoughts?
I agree with Joanie. We're in a big crunch right now and won't have cycles to think about this properly in time for the OOo deadline. When the GNOME 2.26 and OpenSolaris 2009.06 deadlines blow over, though, maybe we should set up a conference call to go over this and just focus on trying to get it into some future version of OOo.
Since it doesn't make sense to work on this issue without the input from AT people (Orca), who also need to enhance AT for this, I will make an exception and "downtarget" an A11y OOo 3.2 issue...
Trackback to the Orca bug: http://bugzilla.gnome.org/show_bug.cgi?id=525895
Adjusting target to OOo 3.3 It is time to focus on this issue in order to get it solved for OOo 3.3. The release schedule of OOo 3.3 can be found at http://wiki.services.openoffice.org/wiki/OOoRelease33. Thus, our deadline is more or less the beginning of March 2010. I propose to have the details on how annotations/notes are exposed ready until the beginning of January 2010. Then enough time is left to also reflect on the first implementation and to adjust it, if needed. Joanie and Willie: Please provide your input. Malte: May be you should contact Pete Brunet for his input.
Ping. Any input from the stakeholders?
I'm looking at this attachment from mt: ~~~~~~~~~ Created an attachment (id=58829) Sample document with annotation, visible in OOo 3, not in OOo 2 ~~~~~~~~~ Based on this sample, it seems that an annotation: * is a scrollable text object whose color can vary * contains paragraphs whose text can be formatted * contains a text (paragraph?) object (or two?) to display the commenter and the time of the comment. * contains a button-like object with an associated pop-up menu According to Accerciser, the hierarchy starting from the document frame is, in its entirety: -> ATK_ROLE_DOCUMENT_FRAME -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH The second note is associated with the 8th paragraph. Therefore, just considering the second note, I'd expect the hierarchy to instead be something *like*: -> ATK_ROLE_DOCUMENT_FRAME -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_SCROLL_PANE -> ATK_ROLE_PARAGRAPH ("Annotation") -> ATK_ROLE_PARAGRAPH ("1st paragraph bold") -> ATK_ROLE_PARAGRAPH ("2nd paragraph italic") -> ATK_ROLE_PARAGRAPH ("3rd paragraph centered") -> ATK_ROLE_{TEXT,PARAGRAPH} ("Oliver-Rainer Wittmann") -> ATK_ROLE_{TEXT,PARAGRAPH} ("12/12/2008 15:01") -> ATK_ROLE_PUSH_BUTTON -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH -> ATK_ROLE_PARAGRAPH I'd expect to be able to identify the position of note by your implementation of AtkComponent. (Related aside: atk_component_grab_focus might be a handy, alternative way for an AT to deal with the issue of how to move focus there efficiently.) I'd expect all of the text objects to implement AtkText and, if editable, AtkEditableText; to expose the associated text attributes; to reflect the appropriate states; etc., etc. (I'd also expect these items to be fully keyboard navigable, but that's a different issue.) So the above is a long way of saying that I personally don't believe annotations are anything really new a11y-wize. They're seem to me to be like fancy little embedded documents. :-) And you're already implementing a11y support for documents. So could you do essentially what you're already doing there, but make the annotations be children of the paragraph with which they are associated? Beyond the above: * I believe that forms can contain scroll panes as well. It would be nice to have a reliable, easy way for an AT to be able to distinguish the two. I think this could be an object attribute. * It might be worth doing something to suggest what the underlying roles are with respect to the text within an annotation (e.g. Author, Date, note content, etc.). This might be another candidate for object attributes. Perhaps the ARIA specifications and implementation by browsers could serve as a model?? * I wonder... Would it be worth implementing ATK_RELATION_FLOWS_{TO,FROM} so that one annotation could point to the {next,previous} annotation? That might also be a handy way for an AT to address the issue of efficient navigation amongst these items. * We can already identify where the little triangle at the beginning of the range is from the embedded object character which is present in the AtkText implementation. However, as MT observed: ~~~~~~~~~ - With ODF 1.2 they can be bound to a range starting in one paragraph, ending in an other - how to expose this relation? ~~~~~~~~~ Ugh. I'm not sure on this one. How would I as a sighted user notice/identify this?
OD->joaniediggs: Thanks for your input. I will soon setup a corresponding OOo wiki page to refine the details how comments are exposed to ATs. Please stay tuned for my comments and questions. By the way, the 'notes' are renamed in OOo 3.2 to 'comments'. In the ODF they are named 'annotations'.
OD->joaniediggs: I have setup the promised OOo wiki page - http://wiki.services.openoffice.org/wiki/Writer/Accessibility. I have copied your proposed solution - please have a look, if it is ok. I propose to use your proposed solution as the starting point to work out together the to be implemented stuff for this issue. My comments/questions: - I am right that the ATKObject with role ATK_ROLE_SCROLL_PANE in the proposed hierarchy is the representative of the comment? - To distinguish a comment scroll pane from other scroll panes you are proposing a new attribute. You are also suggesting to have a certain attribute to identify the role of the comment's children. Are you thinking of a new general type attribute? Something like ATKAttribute{ "type", "comment"|"comment content"|"comment author"|"comment date"|"comment actions" }? - A relation between the comments should be implemented if such a relation is also exposed in the user interface. In my opinion the ATs should not 'see' more than the normal user. - For a comment which annotates a certain text range of the document content also the end position is available. Currently, it is not yet been decided, if it will be present as a character in the text string. I propose that the accessibility object which represents such a comment is related with the paragraph which contains the start of the annotated text range. May be a new attribute at the comment representing accessibility object is needed which contains the end position. We have also not decided yet how the annotated text range is presented in the user interface - may be something like a selection is presented in the user interface. - What about the comments which are not visible in the user interface, because -- all comments are hidden by triggering the corresponding user action (Menu View - Comments) OR -- there are too much comments for the text on a page? I propose that these comments are also not visible for the ATs. Ok? - What about the buttons to bring certain comments into view when too much comments are present?
joaniediggs->OD: - I am right that the ATKObject with role ATK_ROLE_SCROLL_PANE in the proposed hierarchy is the representative of the comment? Yes. - To distinguish a comment scroll pane from other scroll panes you are proposing a new attribute. You are also suggesting to have a certain attribute to identify the role of the comment's children. Are you thinking of a new general type attribute? Something like ATKAttribute{ "type", "comment"|"comment content"|"comment author"|"comment date"|"comment actions" }? Something along those lines, yes. I don't have a specific set of attribute names and values in mind; I would just like a reliable way to identify these objects. - A relation between the comments should be implemented if such a relation is also exposed in the user interface. In my opinion the ATs should not 'see' more than the normal user. Fine with me. - For a comment which annotates a certain text range of the document content also the end position is available. Currently, it is not yet been decided, if it will be present as a character in the text string. I propose that the accessibility object which represents such a comment is related with the paragraph which contains the start of the annotated text range. Makes sense. -(con't) May be a new attribute at the comment representing accessibility object is needed which contains the end position. Perhaps. Although I think this will be tricky. Each paragraph is exposed to us as a separate AtkObject. So the end position is not merely an offset; it's an offset in an accessible object, which may or may not be the same object as the one in which the annotation started. I have no idea what the right thing to do is. Merely pointing out a potential issue. -(con't) We have also not decided yet how the annotated text range is presented in the user interface - may be something like a selection is presented in the user interface. I'm not sure. Perhaps Will could chime in on this (and, for that matter, the other issues). - What about the comments which are not visible in the user interface, because -- all comments are hidden by triggering the corresponding user action (Menu View - Comments) OR -- there are too much comments for the text on a page? I propose that these comments are also not visible for the ATs. Ok? Maybe. :-) Is it possible for the user to arrow to a block of text which has an associated comment, but that comment is not visible on the screen? If so, then not exposing that comment to ATs will be problematic. If, on the other hand, moving to a block of text which has an associated comment necessarily causes that comment to be visible on the screen (and thus visible to ATs), I have no objections to what you propose. - What about the buttons to bring certain comments into view when too much comments are present? They should be exposed as push buttons. Although this raises an issue I'd not really given much thought to -- and don't know what the answer is: My proposal is very much based on users who are blind. The user would presumably be reading the text by arrowing through it. When there is an comment present, the screen reader should at the very least be able to: 1. announce the presence of the comment 2. provide an easy way to access the comment's contents 3. provide an easy way to return to the text being read Exposing a comment as a child of the text to which it applies makes it easy for the screen reader to do the above. And is consistent with the flow of the content. The fact that all of the comments exist off to the right of the document and that there are buttons which can be used to scroll comments into/out of view is, I believe, largely irrelevant to the user who is blind. But it might be very relevant to the user who is sighted but requires an alternative input device. It would be good if someone more familiar with that area could chime in as that is not my background.
OD->joaniediggs: Sorry for the long silence. I had a deeper look at the used windows and controls which are used to present a comment in the user interface. I also prototyped some stuff in order to evaluate different solutions to expose the current user interface presentation to accessibility tools. Please have a look at my refined proposal in the wiki - see http://wiki.services.openoffice.org/wiki/Writer/Accessibility There some minor changes regarding your proposed document hierarchy. I am proposing to use the accessible description to identify the different elements - it's easy to implement and no new attributes are needed. In general I think that a lot of stuff needs to be clarified regarding how comments shall be exposed to accessibility tools. Thus, I am proposing to implement in OOo a first version of exposing comments to accessibility tools. Its usage in the accessibility tools will show how it works, what is missing, what needs to be changed and what needs to be corrected. Thus, we will get input for a next version of exposing comments to accessibility tools and can implement this next version for a following release of OOo. Do you agree on this approach?
There is a lot to process in this thread with a number of very long replies to read. I apologize for adding to the length. :-( @ww->od: "I am proposing to use the accessible description to identify the different elements - it's easy to implement and no new attributes are needed." The accessible description is for human consumption and is not really meant to provide any meaning or direction for assistive technologies -- it's just a string to pass on to users. In looking at what you are proposing to expose to the AT, most of the content is probably just inferable to a user, just as a sighted user can infer the content type. That is, a date looks like a date, a name looks like name, etc. For the button that opens a menu of further actions, giving it an accessible name of something like "Actions" and an accessible description of "Activate this button to open a list of actions to perform on this comment" might be enough for the user to infer its purpose. The accessible description should also appear as a tooltip that is available when hovering over the button and/or when one presses Ctrl+F1 when the button has keyboard focus. What would be very nice is for the user to know that the entire container is a comment. ARIA has the notion of a "note" role: http://www.w3.org/TR/wai-aria/roles#role_definitions. I believe you can expose this via the "xml-roles" attribute of the container for the object. See https://developer.mozilla.org/ARIA_User_Agent_Implementors_Guide. Popping up a level, the other big questions seem to be: 1) Representing the hierarchy 2) Identifying the range(s) of text being commented on 3) Handling visibility of comments 4) Providing keyboard-only access to comments - navigating to them, editing them, copying them, creating new ones, etc. Representing the hierarchy -------------------------- The current proposal is to embed an EOC character in the paragraph and then make the container for the comment object be a child of the paragraph. The position of the EOC character represents the start of the range(s) of text being commented on, and the container hierarchy basically maps to the building blocks for the comment. This seems fine to me. Identifying the range(s) of text being commented on --------------------------------------------------- This is where things get a little messy. In all the examples I've seen so far, comments seem to be anchored to a single point in the text. But, http://lists.oasis-open.org/archives/office/200708/msg00007.html indicates some sort of new <office:annotation-end> tag. I'm not sure how to interpret this, but it seems to imply a comment can apply to a range of text instead of a single point, and that comments are represented by either one or a pair of objects. If it's one object, then it's a single point in the document. If it's two, the first is the start of a range and the second is the end of that range. Assuming we use the EOC model as it exists, the first object can always get us to the actual comment container directly. That leaves the second object. To handle that, I propose the following -- the second object is embedded as an EOC just like the first, but it is basically a non-showing empty object whose xml-role is "note-end". The link between the start and end objects can be handled via an accessible relation RELATION_MEMBER_OF added to both objects. With this, when I encounter the first object, it has the full container with an xml-role attribute of "note". It also has a RELATION_MEMBER_OF relation that points to the end object (if it exists). When I encounter the end object, it is basically empty (maybe ROLE_FILLER?), but with an xml-role attribute of "note-end". It also has a RELATION_MEMBER_OF relation that points to the starting object. Handling visibility of comments ------------------------------- IMO, these are like any other objects and the STATE_SHOWING state should be used to indicate whether they are burning pixels on the screen or not. Providing keyboard-only access to comments ------------------------------------------ I'm not really familiar with all the keyboard navigation techniques and styles in OOo, so I'll leave this up to the OOo developers. From a user-based task model, however, the important things seem to be: * The user should be able to quickly add or delete a comment. * The user should be able to quickly navigate between the document content and the comments. * When in the document content area and the user sees a comment is present, navigating to the comment should automatically make the comment become visible. I might also argue that as you arrow through a document, any associated comment for the current caret position should be visible and it should be obvious which comment it is. * The user should be able to quickly jump from one comment to the next (e.g., let them quickly scan what other people had to say about the document). * The user should be able to easily navigate within a comment -- go from field to field, select/copy/edit text, etc.
OD->ww: Thanks for your input. I will sum up the requirements for exposing the comments to accessibility tools on the wiki page in order to have something like a ToDo list for the implementation. Here we can also decide which requirements shall be fulfilled for the first version and which can be postponed. By the way, ARIA role "note" from http://www.w3.org/TR/wai-aria/roles#role_definitions is not mentioned on https://developer.mozilla.org/ARIA_User_Agent_Implementors_Guide. But, I agree that it makes sense to introduce roles "note" and may be "note end". Some words about comments which annotate a certain range of a document: I like the idea of having a non-showing accessible object for the end position of a comment and a relation to the accessible object of the intrinsic comment. I am not sure, if there will be special character in the text marking the end position of a comment. By the way, I am not sure about term "EOC character". What does it mean?
joaniediggs->OD: EOC == "embedded object character". Will and I have had to deal with (and communicate about) them so much in Gecko that we've gotten lazy and started abbreviating. :-)
OD->joaniediggs,ww: As promised I have summarized the requirements on the wiki page. Please have a look and give your feedback.
ww->od: I think the proposal on the WIKI looks good. The only thing I see as an issue is that the NOTE role does not exist in ATK. What you may need to do is keep the AT-SPI role as it was and then define an 'xml-roles' attribute that has a value of 'note' and store this in the accessible object's 'attributes' property. The way this would be exposed to the assistive technology is via the 'attributes' property on the accessible object. Note that the 'attributes' property on the accessible object is a whole different space than the 'attributes' property associated with the accessible text for the object. That it, there are two 'attributes' like things in the API - one is for general attributes on the whole object (which is where we want to put the 'xml-roles' attribute) and the other is for text attributes. Thanks!
od->ww: Thanks for the review and the feedback. I will adjust the ToDo list in the wiki: ad requirement 1.: Identification of the uppermost accessible object by OOo's accessible role "COMMENT". In OOo's ATK bridge this accessible role should be mapped to ATK role ATK_ROLE_SCROLL_PANE and new AtkAttribute { "xml_roles", "note" }. The AtkAttribute is provided via ATK-method <AtkObject::atk_object_get_attributes(..)> ad requirement 7.: [mapping statement for OOo's ATK bridge as for 1.]
od->ww, joaniediggs: One more question: For the non-showing accessible object marking the end position of a text range which is annotated by a comment I choose ATK role ATK_ROLE_UNKNOWN. Is this ok for you?
ww->od: I think UNKNOWN is OK. I'm not sure there is another role that might make sense. Thanks!
correction of <AccessibleEditableTextPara> - fallback to parent's <XAccessibleContext> for method <getLocationOnScreen()> - changed file: /svx/source/accessibility/AccessibleEditableTextPara.cxx, change set 50438b17f5b0 refactoring of sidebar window code used for visualization of comments/annotations: change sets 9685e8fd8c1a and 554869494a1d
implementation of first part of accessible Writer comments/annotations ready - change set 8f96562d116e
Implementation of second and third part of accessible Writer comments/annotations ready - change sets a2196142655e, 9d5a2f00014e and 9c07b56818cf The implementation is more or less ready - see corresponding wiki page http://wiki.services.openoffice.org/wiki/Writer/Accessibility. Testing and review are needed now.
Upps, not yet ready - two requirements are still not solved ;-) - implementation continues
Finishing implementation of accessible Writer comments - change set fe3bc3fac16a od->willie, joanie: Please have a look at the wiki page (http://wiki.services.openoffice.org/wiki/Writer/Accessibility). Most of the requirements have been solved in cws sw33a11y01. Some requirements are adjusted or removed.
OD->ES: checked in internal installation set of cws sw33a11y01 - please verify. Please have a look at wiki page http://wiki.services.openoffice.org/wiki/Writer/Accessibility for your testing. OD->willie, joanie: I will notify you when cws sw33a11y01 is integrated into DEV300 milestone and when the corresponding developer snapshot is available.
Verified in CWS sw33a11y01
Closing accessibility issues which have been fixed, verified and integrated...